Skip to content

fix(manager): optimize video coverage query (22-47s → <1s)#637

Merged
tataihono merged 45 commits intomainfrom
fix/manager-coverage-query-performance
Apr 2, 2026
Merged

fix(manager): optimize video coverage query (22-47s → <1s)#637
tataihono merged 45 commits intomainfrom
fix/manager-coverage-query-performance

Conversation

@tataihono
Copy link
Copy Markdown
Contributor

Summary

  • Root cause: Manager's /api/videos fetched 414K variant rows + 20K subtitle rows through Strapi GraphQL to compute per-video coverage. This took 22-47s per cache refresh, blocking auth checks and causing session timeouts.
  • Fix: New CMS REST endpoint (/api/video-coverage) computes coverage via SQL aggregation (60-660ms). Manager calls this instead of GraphQL.
  • Also: Reverts maxLimit: 100 from fix(cms): fix production PG pool exhaustion causing KnexTimeoutErrors #626 (broke pageSize: 5000 pagination), increases language cache TTL from 5min to 24h.

Changes

File Change
apps/cms/src/api/video-coverage/ New REST endpoint with SQL aggregation
apps/cms/config/plugins.ts Revert maxLimit: 100 regression
apps/manager/src/app/api/videos/route.ts Replace GraphQL fetch with REST call
apps/manager/src/features/coverage/coverage-report-client.tsx Consume { human, ai, none } counts
apps/manager/src/app/api/languages/route.ts TTL 5min → 24h

Benchmark (production data)

Query Before After
Global coverage (all languages) 22-47s 660ms
Single language filter 22-47s 60ms

Test plan

  • CMS builds and starts with new endpoint
  • /api/video-coverage returns all ~1083 videos with coverage counts
  • /api/video-coverage?languageIds=529 returns English-filtered counts
  • Manager dashboard loads without logging out
  • Language filter changes reflect updated coverage
  • Report type switching (subtitles/audio/meta) still instant

Post-Deploy Monitoring & Validation

  • What to monitor: Railway logs for @forge/cms and @forge/manager
  • Logs to watch: [video-cache] and [api/videos] log lines
  • Expected healthy behavior: No more KnexTimeoutError or Cannot return null errors. Cache refresh logs show <2s. No Internal Server Error flood.
  • Failure signal: [video-cache] Background refresh failed in manager logs → rollback
  • Validation window: 30 minutes post-deploy
  • Validation query: curl /api/video-coverage | jq '.videos | length' should return ~1083

Compound Engineering
🤖 Generated with Claude Opus 4.6 (1M context) via Claude Code

…sion

- Add /api/video-coverage custom endpoint with SQL aggregation for
  per-video coverage counts (subtitles + audio, human vs AI)
- Revert maxLimit: 100 from GraphQL config (broke pageSize: 5000,
  caused 11-page sequential fetching instead of 1)
- Increase language cache TTL from 5min to 24h (geo data rarely changes)

The new endpoint benchmarks at 60-660ms vs 22-47s for the current
GraphQL approach of fetching 414K variant rows through the ORM.
Replace the 414K-row GraphQL fetch (22-47s) with a call to the new
/api/video-coverage CMS endpoint (60-660ms SQL aggregation).

- Coverage counts { human, ai, none } per video per coverage type
- Collections show own coverage, standalone videos returned separately
- Language-filtered requests hit CMS directly (fast enough at ~60ms)
- Global requests use SWR cache (2min TTL)
- Remove GraphQL query, fetchAllPages, and JS-side coverage computation
Update CmsVideo/CmsCollection types to use { human, ai, none } counts
instead of single status strings. Derive CoverageStatus from counts
via countsToStatus() for existing UI components.

Note: pre-existing eslint rule definition errors (react-hooks/set-state-in-effect,
@next/next/no-img-element) unrelated to this change.
@railway-app
Copy link
Copy Markdown

railway-app bot commented Apr 2, 2026

🚅 Deployed to the forge-pr-637 environment in forge

Service Status Web Updated (UTC)
@forge/manager 🕒 Building (View Logs) Apr 2, 2026 at 3:06 am
@forge/cms ✅ Success (View Logs) Apr 2, 2026 at 1:58 am
1 service not affected by this PR
  • @forge/web

@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 01:54 Destroyed
The collection (e.g. feature film, series) now appears as the first
tile in its grid, showing its own coverage status. Distinguished from
children by a thicker border and rounded corners. In expanded view,
labeled with "(collection)" suffix.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 01:57 Destroyed
…ndalone

Collections are now sorted alphabetically by title in the API response.
Standalone videos group is always last and no longer injects a synthetic
collection tile as the first square.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:02 Destroyed
Tile hover now shows: "Title — Human (2 human, 1 AI, 0 none)"
instead of just "Title — Human".
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:05 Destroyed
Display "X human, Y AI, Z none" counts in the bottom hover bar
next to the status label. Subtle styling to not overwhelm.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:07 Destroyed
Replace single status label with colored pills:
- Green: "X Verified subtitles" (only if > 0)
- Purple: "X AI subtitles" (only if > 0)
- Red: "X No subtitles" (only if > 0)

Label adapts to active report type (subtitles/audio/meta).
In global mode (no language filter), "none" count is computed as
total languages minus human minus AI.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:12 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:13 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:16 Destroyed
The pill X button was only updating draft state without navigating,
so removing a language didn't trigger a data refetch. Now it updates
both the draft state and the URL params immediately.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:18 Destroyed
ModeToggle now renders for subtitles, audio, and meta report types.
Translate button is greyed out with "Coming soon" tooltip on audio
and meta. Switching to audio/meta while in translate mode auto-resets
to explore.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:22 Destroyed
Replace native title attr (doesn't work on disabled buttons) with a
CSS ::after pseudo-element tooltip that appears above the button on
hover showing "Coming soon".
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:24 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:24 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:25 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:51 Destroyed
Native <select> styling is inconsistent across browsers. Replace with
a custom button+menu dropdown following the ReportTypeSelector pattern.
Closes on Escape or outside click.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:52 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:53 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:54 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:54 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:55 Destroyed
Filter collections by type (Series, Feature Film, Standalone, etc.)
using a dropdown next to the coverage segment filter. Standalone is
included as an option. Clear filters resets all three filters.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 02:58 Destroyed
Replace plain "No videos match this filter" text with a centered
empty state showing a search icon, "No results found" title, and
a hint to adjust search/filters.
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 03:02 Destroyed
@railway-app railway-app bot temporarily deployed to forge / forge-pr-637 April 2, 2026 03:02 Destroyed
@tataihono tataihono merged commit b26b425 into main Apr 2, 2026
27 of 28 checks passed
@tataihono tataihono deleted the fix/manager-coverage-query-performance branch April 2, 2026 03:17
tataihono added a commit that referenced this pull request Apr 2, 2026
- Add eslint-plugin-react-hooks v7 at root to resolve
  set-state-in-effect rule used in inline eslint-disable comments
- Register plugin in eslint.config.mjs for tsx/ts files
- Fix prettier formatting lost during #637 squash merge
- Add CLAUDE.md rule: never skip pre-commit hooks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
tataihono added a commit that referenced this pull request Apr 2, 2026
- Add eslint-plugin-react-hooks v7 and @next/eslint-plugin-next at root
  so local eslint matches CI (resolves set-state-in-effect, no-img-element)
- Register both plugins in eslint.config.mjs for tsx/ts files
- Fix prettier formatting lost during #637 squash merge
- Restore eslint-disable comment for no-img-element on <img> tag
- Add CLAUDE.md rule: never skip pre-commit hooks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
tataihono added a commit that referenced this pull request Apr 2, 2026
The root eslint config was missing react-hooks and @next/next plugins
that CI has via eslint-config-next. This caused eslint-disable comments
referencing set-state-in-effect and no-img-element to error locally as
"Definition for rule not found", blocking all commits touching manager.

- Add eslint-plugin-react-hooks v7 and @next/eslint-plugin-next to root
- Register plugins in eslint.config.mjs for tsx/ts files
- Fix prettier formatting lost during #637 squash merge
- Add CLAUDE.md rule: never skip pre-commit hooks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
tataihono added a commit that referenced this pull request Apr 2, 2026
The root eslint config was missing react-hooks and @next/next plugins
that CI has via eslint-config-next. This caused eslint-disable comments
referencing set-state-in-effect and no-img-element to error locally as
"Definition for rule not found", blocking all commits touching manager.

- Add eslint-plugin-react-hooks v7 and @next/eslint-plugin-next to root
- Register plugins in eslint.config.mjs for tsx/ts files
- Fix prettier formatting lost during #637 squash merge
- Add CLAUDE.md rule: never skip pre-commit hooks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
tataihono added a commit that referenced this pull request Apr 2, 2026
* fix(manager): fix lint errors and add no-verify CLAUDE.md rule

Remove eslint-disable/enable comments referencing undefined rules
(react-hooks/set-state-in-effect, @next/next/no-img-element) and
fix blank lines left behind. Add CLAUDE.md rule to never skip hooks.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: add react-hooks and next eslint plugins to root config

The root eslint config was missing react-hooks and @next/next plugins
that CI has via eslint-config-next. This caused eslint-disable comments
referencing set-state-in-effect and no-img-element to error locally as
"Definition for rule not found", blocking all commits touching manager.

- Add eslint-plugin-react-hooks v7 and @next/eslint-plugin-next to root
- Register plugins in eslint.config.mjs for tsx/ts files
- Fix prettier formatting lost during #637 squash merge
- Add CLAUDE.md rule: never skip pre-commit hooks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant